Papers with transformer encoder
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-modal entity alignment (MMEA) is a critical task that aims to identify equivalent entity pairs across multi-modal knowledge graphs (MMKGs). |
| Approach: | They propose a novel MMEA transformer that hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance alignment task. |
| Outcome: | The proposed transformer hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance the alignment task. |
A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products (2022.coling-1)
Copied to clipboard
| Challenge: | Existing pre-trained language models lack medicinal product knowledge for product vertical search. |
| Approach: | They propose a biomedical knowledge enhanced pre-trained language model for medicinal product vertical search using ELECTRA’s replaced token detection (RTD) pre-training. |
| Outcome: | The proposed model improves query-title relevance, query intent classification, and named entity recognition in query. |
Long Document Summarization with Top-down and Bottom-up Inference (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent models infer latent representations of words or tokens with a transformer encoder, which is bottom-up and thus does not capture long-distance context well. |
| Approach: | They propose a method to infer latent representations of words or tokens in documents . they assume a hierarchical structure of a document where top-level captures long range dependency . |
| Outcome: | The proposed model can summarize an entire book and achieve competitive performance on a wide range of document summarization benchmarks. |
“All that Glitters”: Techniques for Evaluations with Unreliable Model and Human Annotations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using standard metrics in the presence of poor labels masks label and model quality . evaluation techniques accounting for unreliable labels reveal important flaws, including spurious correlations and nonrandom racial biases . |
| Approach: | They analyze human labels, GPT model ratings, and transformer encoder model ratings . they show that standard metrics in the presence of poor labels mask label and model quality . |
| Outcome: | The proposed methods mask label and model quality even in the presence of poor models. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions (2020.acl-main)
Copied to clipboard
| Challenge: | a growing demand for the ability to communicate in English means automated tutoring and assessment systems are becoming more popular. |
| Approach: | They propose to use automatic speech recognition transcripts to grade spontaneous speech based on textual features. |
| Outcome: | The proposed system improves on a transformer encoder with native language identification as an auxiliary task. |
Porous Lattice Transformer Encoder for Chinese NER (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to integrate word boundary information into character-level Chinese NER are inefficient and lack semantic interaction. |
| Approach: | They propose an extension of transformer encoder that is tailored for ChineseNER to incorporate lexicons into character-level Chinese NER by lattices. |
| Outcome: | The proposed extension performs 11.4 times faster than state-of-the-art methods while retaining the rich long-term dependencies. |
Multi-Hop Question Generation via Dual-Perspective Keyword Guidance (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work fails to fully utilize the guiding potential of keywords and neglect to differentiate the distinct roles of question-specific and document-specific keywords. |
| Approach: | They propose a dual-perspective keyword-guided framework that integrates question and document keywords into the multi-hop question generation process. |
| Outcome: | The proposed framework integrates question and document keywords into the multi-hop question generation process. |
Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Aspect-based sentiment classification is a popular task aimed at identifying the corresponding emotion of a given aspect. |
| Approach: | They propose a dependency graph enhanced dual-transformer network to support mutual reinforcement between the flat representation learning and graph-based representation learning. |
| Outcome: | The proposed model outperforms state-of-the-art methods on five datasets with a large margin. |
Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings that consider context have attracted great attention for natural language processing tasks in recent years. |
| Approach: | They propose two different approaches to integrate contextualized word embeddings with transformer encoders for sentence similarity modeling. |
| Outcome: | The proposed model outperforms the feature-based approach on six datasets. |